---
layout: post
title: Why using HDF5?
date: '2015-01-01T11:44:00.000-08:00'
author: Alex Rogozhnikov
tags:
- Machine Learning
- Optimization
modified_time: '2015-01-02T12:43:26.234-08:00'
blogger_id: tag:blogger.com,1999:blog-307916792578626510.post-2272022213403979655
blogger_orig_url: http://brilliantlywrong.blogspot.com/2015/01/why-using-hdf5.html
---

<div dir="ltr" style="text-align: left;" trbidi="on">
  <b>Update</b>. Everything below is inessential, since I've found the <a href="http://stackoverflow.com/a/27713489/498892">stackoverflow</a> answer about hdf5.<br /><br />
  The only thing &nbsp;don't agree with is blaze, I've tried it and it is obviously raw and needs much time to become not even stable but at least really useful.<br /><br /><br />
  My current workflow is completely based on IPython, and I'm working much with pandas (which I personally consider as a good example of poor library design).
  <br /><br />Nevertheless, &nbsp;I moved recently to HDF, though installing pyTables (which is needed to use hdf with pandas) isn't as straightforward as I expected.<br /><br />
  And now I convert all the data to hdf.<br /><br />
  <ul style="text-align: left;">
    <li>First, this usually results in less (about 2-3 times) space needed to store data (but that depends on the dataset, for some ot them there is no difference between csv and hdf).</li>
    <li>Second, your data now stored in binary format, so all the types are strictly defined - no parsing is needed, no guessing of types</li>
    <li>Thus, read/write operations are orders of times faster</li>
    <li>And no approximations are made for float numbers.</li>
  </ul>
  <div>Hope, there are enough arguments to move to hdf.</div>
</div>
